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USER INTERFACE FOR INTERACTIVE REMOVAL OF 
DEFECTS IN AN IMAGE SEQUENCE 

Field of the Invention 

5 The present invention relates to digital image 

restoration techniques, and more particularly, to a user 
interface that facilitates the removal of defects in an image. 

Background of the Invention 

10 Digital motion picture restoration involves a variety 

of image processing operations that require a considerable amount 
of computations and take a relatively long amount of time, even 
with powerful computers. An example of a computation- intensive 
image processing technique is motion estimation, which is 
Jl5 required for high fidelity repair. Generally, motion estimation 
f : i allows a defective region of an image to be replaced with a 
m corresponding image region from a previous or subsequent image 
!*f frame . 

M, Longer processing times are acceptable if the 

IJZO computations are conducted automatically, without user 
s p interaction. Numerous complex algorithms have been developed for 
*~ automatic image sequence restoration. For a detailed discussion 
r| of such image restoration techniques, see, for example, Anil 
f ** Kokaram, "Motion Picture Restoration," (Springer, 1998). 
25 Currently available image restoration techniques, however, do not 
provide a complete solution. Generally, currently available 
image restoration techniques suffer from incomplete repair or 
false detection (or both) , resulting in image artifacts. Thus, 
human intervention is eventually required to complete the image 
30 restoration and obtain high quality artifact-free restoration. 

Image restoration techniques typically require an 
operator to work with the original image sequence and the 
automatically repaired image sequence in an interactive process 
until the operator is satisfied with the restoration. Generally, 
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the operator reviews the repaired image sequence and accepts, 
rejects or further modifies each restored image portion. This 
workflow has certain disadvantages, which if overcome, could 
greatly improve the efficiency and quality of the restoration 
5 process. For example, currently available image restoration 
systems do not provide a clear indication of areas that have been 
repaired automatically in current or previous steps. Thus, the 
operator must carefully investigate every frame in order to 
determine whether any artifacts were introduced. In addition, 
10 current systems do not allow a defect that appears in a number of 
consecutive frames, for example, caused by dirt on the camera 
lens, to be repaired in an efficient manner. 

A need therefore exists for a method and apparatus for 
removing defects from images that provides a better user 
Bl5 interface for image sequence repair. A further need exists for a 
JJi method and apparatus for removing defects from images that 
O provides an improved mechanism for repairing a defect that 
L. appears in a number of consecutive frames. 

jpO Summary of the Invention 

^ Generally, a method and apparatus are disclosed for 

F\ repairing an image that provides an improved user interface that 
^ allows a user to modify an image defect matte indicating portions 
of an original image that have been repaired in previous 
25 iterations or are proposed modifications for a current iteration. 
The user interface incorporates graphical tools that allow the 
user to modify or manipulate the defect matte and thereby 
initiate commands that control the selection, detection or repair 
of one or more defects in an image. For example, the user 
30 interface may include commands that allow a user to further 
repair or cancel a previous repair of a selected region of an 
image . 
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Each image defect matte may be stored, for example, as 
an array of elements that assume integer values. For example, 
the integer values may be mapped to a limited set representing 
different degrees or steps of repair. For example, the set of 
5 possible matte values can include "original" (unmodified) , 
"repaired" (automatically or manually in a previous step) and 
"new defect" (marked for further repair) . Each of the matte 
integer values can be encoded with a different color. Each 
element of the defect matte corresponds to one or more image 
10 pixels. Thus, the defect matte can be projected onto the 
original or repaired image to facilitate the repair process. 

The disclosed image defect removal system can also 
determine a defect metric for each image defect matte that 
^ quantifies the extent of the defects in the corresponding image. 
Ki5 The defect metric can indicate, e.g., the number of defects in a 
m given image defect matte or a sum of the image area having a 
O defect. In this manner, the defect metric can be employed to 
L prioritize the images in need of repair or to traverse each image 
s frame having a defect metric above a predefined threshold. 
3>0 A more complete understanding of the present invention, 

H s as well as further features and advantages of the present 
?! invention, will be obtained by reference to the following 
"* r ~ detailed description and drawings. 

25 Brief Description of the Drawings 

FIG. 1 illustrates an image defect removal system in 
accordance with the present invention; 

FIG. 2A illustrates an original image that may be 
repaired in accordance with the present invention; 
30 FIG. 2B illustrates an image defect matte indicating 

portions of the original image of FIG. 2A that have been repaired 
in accordance with the present invention; 
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FIG. 2C illustrates the original image of FIG. 2A 
superimposed with the image defect matte of FIG. 2B; and 

FIG. 3 is a flow chart describing an exemplary image 
repair process embodying principles of the present invention. 

5 

Detailed Description 

FIG. 1 illustrates an image defect removal system 100 
in accordance with the present invention. As shown in FIG. 1, 
the image defect removal system 100 includes certain standard 
10 hardware components, such as a processor 110 and a data storage 
device 120, such as a read-only memory and/or a random access 
memory (RAM) . The data storage device 12 0 includes one or more 
original images 200, discussed below in conjunction with FIG. 2A, 
% i one or more image defect mattes 250, discussed below in 
6(15 conjunction with FIG. 2B, and an image repair process 3 00, 
discussed below in conjunction with FIG. 3. Generally, the image 
O repair process 3 00 employs a user interface in accordance with 
i2 the present invention that indicates image areas that have been 
% repaired or are proposed for repair and allows a user to initiate 

"SO commands that control the selection, detection or repair of one 

or more defects in an image, 
pii The methods and apparatus discussed herein may be 

M distributed as an article of manufacture that itself comprises a 
computer readable medium having computer readable code means 
25 embodied thereon. The computer readable program code means is 
operable, in conjunction with a computer system, to carry out all 
or some of the steps to perform the methods or create the 
apparatuses discussed herein. The computer readable medium may be 
a recordable medium (e.g., floppy disks, hard drives, compact 
30 disks, or memory cards) or may be a transmission medium (e.g., a 
network comprising fiber-optics, the world-wide web, cables, or a 
wireless channel using time-division multiple access, code- 
division multiple access, or other radio- frequency channel) . Any 
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medium known or developed that can store information suitable for 
use with a computer system may be used. The computer- readable 
code means is any mechanism for allowing a computer to read 
instructions and data, such as magnetic variations on a magnetic 
5 media or height variations on the surface of a compact disk. 

Data storage device 120 will configure the processor 
110 to implement the methods, steps, and functions disclosed 
herein. The data storage device 12 0 could be distributed or local 
and the processor 110 could be distributed or singular. The data 
10 storage device 120 could be implemented as an electrical, 
magnetic or optical memory, or any combination of these or other 
types of storage devices. It should be noted that, although data 
storage device 120 is shown separately from other elements of the 
Hi- image defect removal system 100, this is not necessarily the case 
R15 for all applications. Moreover, the term "memory" or Mata 
2* storage device" should be construed broadly enough to encompass 
O any information able to be read from or written to an address in 
[7 the addressable space accessed by processor 110. With this 
*' definition, information on a network is still within data storage 
j§>0 device 120 of the image defect removal system 100 because the 
H processor 110 can retrieve the information from the network. In 
fM a further variation, the image defect removal system 100 may be 
H embodied as an application specific integrated circuit (ASIC) 
that may be incorporated, for example, in a workstation or other 
25 image processing device. 

FIG. 2A illustrates an exemplary original image that 
may be repaired by the image defect removal system 100 in 
accordance with the present invention. As shown in FIG. 2A, the 
image 200 includes one or more objects, such as the object 210, 
30 and one or more defect regions, such as the defect 22 0 
corresponding, for example, to dirt on a camera lens. 

FIG. 2B illustrates an image defect matte 250. As 
previously indicated, the image defect matte 250 indicates 
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portions of an original image that have been repaired in previous 
iterations or are proposed modifications for a current iteration. 
In addition, user interface incorporates graphical tools that 
allow the user to modify or manipulate the defect matte 250 and 
5 thereby initiate commands that control the selection, detection 
or repair of one or more defects in an image. In the exemplary 
image defect matte 250 shown in FIG. 2B, the matte 250 includes a 
region 260 indicating a user-specified region of the original 
image that the user would like to have repaired (or further 
10 repaired) . 

Each image defect matte 250 may be stored, for example, 
as an array of elements that assume integer values. For example, 
the integer values may be mapped to a limited set, representing 
O different degrees or steps of repair. In the exemplary 
f£f5 embodiment, the set of possible values allows each region of an 
JJf image to be marked as "original" (unmodified) , "repaired" 
O (automatically or manually in a previous step) and "new defect" 
(for further repair). Each element of the defect matte 250 
s- corresponds to one or more image pixels. This correspondence 
^0 defines the projection of the defect matte 250 onto the original 
M or repaired image, discussed below in conjunction with FIG. 2C. 
Jj:f In one embodiment, different values of the defect matte 

U are encoded with different colors. In the embodiment shown in 
FIG. 2B, the defect matte 250 includes a boundary 260 drawn 
25 around pixels with the same matte value, such as a given boundary 
around a pixel region that the user has marked as a "new defect" 
for further repair. It is noted that pixels in the repaired 
image that correspond to defect matte elements that have an 
"original" value always coincide with the original image. FIG. 
30 2C illustrates the original image 200 of FIG. 2A superimposed 
with the image defect matte 250 of FIG. 2B. 

FIG. 3 is a flow chart describing an exemplary image 
repair process 300. As previously indicated, the image repair 
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process 3 00 employs a user interface indicating image areas that 
have been repaired or are proposed for repair and allows a user 
to initiate commands that control the selection, detection or 
repair of one or more defects in an image. In the exemplary 
5 embodiment shown in FIG. 3, it is assumed that the user starts 
with a blank matte 250 corresponding to an unmodified original 
image 200. The present invention may also be employed in an 
environment where defects are initially identified and repaired 
using an automated approach. Thus, in such an automated 
10 environment, the defect matte 250 would initially indicate any 
defects identified or repaired by the automatic process, as would 
be apparent to a person of ordinary skill in the art. 

As shown in FIG. 3, the image repair process 300 
k t initially presents the user with the original image 200 or the 
ffl5 repaired image (or both) and optionally the image defect matte 

~ „ 3 

S 250 during step 310. Thereafter, a test is performed during step 
O 32 0 to determine if a user command or modification of the matte 
[I 250 is received. The user command can include, for example, a 
s selection of image areas in the defect matte 250 with a 
■?S0 "repaired" value representing false defects, and changing the 
M selected defect matte elements from "repaired" to "original." In 
JK this manner, corresponding pixels in the repaired image will be 
replaced with pixels from the original image 200. 

In one variation, the user's actions immediately affect 
25 only the defect matte 250 rather than the image itself, and the 
image is not modified until affirmatively initiated by the user. 

If it is determined during step 320 that a user command 
or modification of the matte 250 is not received, then program 
control returns to step 32 0 until such a user command or 
30 modification of the matte 250 is received. 

Once it is determined during step 32 0 that a user 
command or modification of the matte 250 is received, then the 
change indicated by the received command or matte modification is 
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implemented during step 330. Finally, the image 200 and the 
defect matte 250 are updated during step 340 to reflect the 
changes indicated by the received command or matte modification, 
before program control terminates. 
5 In further variations, depending upon context, the 

changing of the defect matte 250 may or may not result in 
subsequent automatic changes of the repaired image. An example 
of an automatic change is canceling a defect or multiple defects. 
An example of a delayed change of the repaired image is repair of 
10 steady dirt that persists for more than one frame. The user may 
easily mark defective areas in subsequent frames. The immediate 
result of marking defective areas will be changes to the defect 
matte 250, with elements of the defect matte changing their 
"*ff values to tt new defect." In such an embodiment, an additional 
jls user command is thereafter required to perform the actual repair. 
W In a further variation, the image defect removal system 

O ,100 can also determine a defect metric for each image defect 
ft matte 250. Generally, the defect metric quantifies the extent of 
3.- the defects in the corresponding image 200. For example, the 
"$0 defect metric can indicate the number of defects in a given image 
H defect matte 250 or a sum of the image area having a defect. In 
% this manner, the defect metric can be employed to prioritize the 
M images in need of repair. In yet another variation, the defect 
metric can be employed as a filter for navigational purposes, 
25 such that a user can easily traverse each image frame having a 
defect metric above a predefined threshold. 

It is to be understood that the embodiments and 
variations shown and described herein are merely illustrative of 
the principles of this invention and that various modifications 
30 may be implemented by those skilled in the art without departing 
from the scope and spirit of the invention. 
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